
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Package Manager &#8212; PixieDust Documentation</title>
    <link rel="stylesheet" href="_static/better.css" type="text/css" />
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    <link rel="stylesheet" href="_static/custom.css" type="text/css" />
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    './',
        VERSION:     '1.0',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true,
        SOURCELINK_SUFFIX: '.txt'
      };
    </script>
    <script type="text/javascript" src="_static/jquery.js"></script>
    <script type="text/javascript" src="_static/underscore.js"></script>
    <script type="text/javascript" src="_static/doctools.js"></script>
    <link rel="shortcut icon" href="_static/pd_icon.ico"/>
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="Use Scala in a Python Notebook" href="scalabridge.html" />
    <link rel="prev" title="Display Data" href="displayapi.html" />
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
  </head>
  <body>
    <header id="pageheader"><h1><a href="index.html ">
        PixieDust Documentation
    </a></h1></header>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <div class="section" id="package-manager">
<h1>Package Manager<a class="headerlink" href="#package-manager" title="Permalink to this headline">¶</a></h1>
<div class="section" id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
<p>You can use the PackageManager component of PixieDust to install and
uninstall maven packages into your notebook kernel without editing
configuration files. This component is essential when you run
notebooks from a hosted cloud environment and do not have access to the configuration files.</p>
<!-- START EXCLUDE --><p>You can use this component in the following ways:</p>
<ul class="simple">
<li>Install a spark package from spark-packages.org.</li>
<li>Install from maven search repository.</li>
<li>Install a jar file directly from an addressable location.</li>
</ul>
</div>
<div class="section" id="install-a-spark-package">
<h2>Install a spark package<a class="headerlink" href="#install-a-spark-package" title="Permalink to this headline">¶</a></h2>
<ol class="arabic">
<li><p class="first">Go to <a class="reference external" href="https://spark-packages.org/">spark-packages.org</a>, and search for your package.</p>
</li>
<li><p class="first">Click the link for your package and locate the code to run the package in spark-shell, pyspark, or spark-submit. For example, you would retrieve the following line:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span>&gt; $SPARK_HOME/bin/spark-shell --packages graphframes:graphframes:0.1.0-spark1.6
</pre></div>
</div>
</li>
<li><p class="first">Copy the maven ID of the package. A maven ID includes a group ID,
artifact ID, and version number, each separated by a colon (:). In
the previous example, the maven ID appears after the packages, such
as <strong>graphframes:graphframes:0.1.0-spark1.6</strong>.</p>
</li>
<li><p class="first">From a Python notebook, enter the following command:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pixiedust</span>
<span class="n">pixiedust</span><span class="o">.</span><span class="n">installPackage</span><span class="p">(</span><span class="s2">&quot;graphframes:graphframes:0.1.0-spark1.6&quot;</span><span class="p">)</span>
</pre></div>
</div>
<p>To fetch the latest release, specify 0 as the version. The PixieDust
packageManager fetches the latest release in the following example.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pixiedust</span>
<span class="n">pixiedust</span><span class="o">.</span><span class="n">installPackage</span><span class="p">(</span><span class="s2">&quot;graphframes:graphframes:0&quot;</span><span class="p">)</span>
</pre></div>
</div>
</li>
</ol>
<p>After you successfully issue the import command, the following results
display:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">Downloading</span> <span class="n">artifact</span> <span class="n">graphframes</span><span class="p">:</span><span class="n">graphframes</span><span class="p">:</span><span class="mf">0.1</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="n">spark1</span><span class="o">.</span><span class="mi">6</span>
<span class="n">Downloaded</span> <span class="mi">326112</span> <span class="n">of</span> <span class="mi">326112</span> <span class="nb">bytes</span> <span class="p">(</span><span class="mf">100.00</span><span class="o">%</span><span class="p">)</span>
<span class="n">Downloaded</span> <span class="n">artifact</span> <span class="n">graphframes</span><span class="p">:</span><span class="n">graphframes</span><span class="p">:</span><span class="mf">0.1</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="n">spark1</span><span class="o">.</span><span class="mi">6</span> <span class="n">to</span> <span class="o">/</span><span class="n">Users</span><span class="o">/</span><span class="n">dtaieb</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">libs</span><span class="o">/</span><span class="n">graphframes</span><span class="o">-</span><span class="mf">0.1</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="n">spark1</span><span class="o">.</span><span class="mf">6.</span><span class="n">jar</span>
<span class="n">Artifact</span> <span class="n">downloaded</span> <span class="n">successfully</span> <span class="n">graphframes</span><span class="p">:</span><span class="n">graphframes</span><span class="p">:</span><span class="mf">0.1</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="n">spark1</span><span class="o">.</span><span class="mi">6</span>
<span class="o">&gt;&gt;&gt;</span><span class="n">Please</span> <span class="n">restart</span> <span class="n">Kernel</span> <span class="n">to</span> <span class="n">complete</span> <span class="n">installation</span> <span class="n">of</span> <span class="n">the</span> <span class="n">new</span> <span class="n">package</span>
<span class="n">Successfully</span> <span class="n">added</span> <span class="n">artifact</span> <span class="n">graphframes</span><span class="p">:</span><span class="n">graphframes</span><span class="p">:</span><span class="mf">0.1</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="n">spark1</span><span class="o">.</span><span class="mi">6</span>
</pre></div>
</div>
<p>Notice the line that instructs you to restart the kernel to complete
installation of the new package. This is required only the first time.
Restart the kernel by using the <strong>Kernel/Restart</strong> menu. After the
kernel is restarted, the library is added to the classpath and can be
used from your Python notebook.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Some libraries, such as GraphFrames include a python module within it. PixieDust automatically adds the python file into the SparkContext. However, you must explicitly call <strong>pixiedust.installPackage</strong> at the beginning of every kernel session so that the python modules are added to the SparkContext.</p>
</div>
<!-- END EXCLUDE --></div>
<div class="section" id="install-from-maven-search-repository">
<h2>Install from maven search repository<a class="headerlink" href="#install-from-maven-search-repository" title="Permalink to this headline">¶</a></h2>
<p>Go to the maven search site, <a class="reference external" href="http://search.maven.org/">search.maven.org</a>, and look for the package of your choice, like <strong>org.apache.commons</strong>. In the results page, open the link of the component you want, like <strong>commons-proxy</strong>. You get the group ID, artifact ID, and version number to use with the <strong>pixiedust.installPackage</strong> API.</p>
<p>By default, PixieDust automatically looks for the following 2 maven repositories: <a class="reference external" href="http://repo1.maven.org/maven2">http://repo1.maven.org/maven2</a> and <a class="reference external" href="http://dl.bintray.com/spark-packages/maven">http://dl.bintray.com/spark-packages/maven</a>. If you use a custom maven repository, you can specify it by using the following base keyword argument:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pixiedust</span>
<span class="n">pixiedust</span><span class="o">.</span><span class="n">installPackage</span><span class="p">(</span><span class="s2">&quot;groupId:artifactId:0&quot;</span><span class="p">,</span> <span class="n">base</span><span class="o">=</span><span class="s2">&quot;&lt;&lt;url to my custom maven repo&gt;&gt;&quot;</span>
</pre></div>
</div>
</div>
<div class="section" id="install-a-jar-file-directly-from-an-addressable-location">
<h2>Install a jar file directly from an addressable location<a class="headerlink" href="#install-a-jar-file-directly-from-an-addressable-location" title="Permalink to this headline">¶</a></h2>
<p>To install a jar file that is not packaged in a maven repository, provide the URL to the jar file. PixieDust will then bypass the maven look up and directly download the jar file from the specified location:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pixiedust</span>
<span class="n">pixiedust</span><span class="o">.</span><span class="n">installPackage</span><span class="p">(</span><span class="s2">&quot;https://github.com/ibm-cds-labs/spark.samples/raw/master/dist/streaming-twitter-assembly-1.6.jar&quot;</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="managing-your-packages">
<h2>Managing your packages<a class="headerlink" href="#managing-your-packages" title="Permalink to this headline">¶</a></h2>
<p>You can uninstall a package by using the following command:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pixiedust</span>
<span class="n">pixiedust</span><span class="o">.</span><span class="n">uninstallPackage</span><span class="p">(</span><span class="s2">&quot;&lt;&lt;mypackage&gt;&gt;&quot;</span><span class="p">)</span>
</pre></div>
</div>
<p>You can also list all the packages installed on your system:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pixiedust</span>
<span class="n">pixiedust</span><span class="o">.</span><span class="n">printAllPackages</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul class="current">
<li class="toctree-l1 current"><a class="reference internal" href="use.html">Use PixieDust</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="install.html">Install PixieDust</a></li>
<li class="toctree-l2"><a class="reference internal" href="loaddata.html">Load Data</a></li>
<li class="toctree-l2"><a class="reference internal" href="displayapi.html">Display Data</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Package Manager</a></li>
<li class="toctree-l2"><a class="reference internal" href="scalabridge.html">Use Scala in a Python Notebook</a></li>
<li class="toctree-l2"><a class="reference internal" href="sparkmonitor.html">Spark Progress Monitor</a></li>
<li class="toctree-l2"><a class="reference internal" href="download.html">Download Data</a></li>
<li class="toctree-l2"><a class="reference internal" href="logging.html">Logging</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="develop.html">Develop for PixieDust</a></li>
<li class="toctree-l1"><a class="reference internal" href="pixieapps.html">PixieApps</a></li>
<li class="toctree-l1"><a class="reference internal" href="pixiegateway.html">PixieGateway</a></li>
<li class="toctree-l1"><a class="reference internal" href="releasenotes.html">Release Notes</a></li>
</ul>

<div id="searchbox" style="display: none" role="search">
  <h3>Quick search</h3>
    <form class="search" action="search.html" method="get">
      <div><input type="text" name="q" /></div>
      <div><input type="submit" value="Go" /></div>
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
  <footer id="pagefooter">&copy; 2017, IBM.
      Created using <a href="http://sphinx-doc.org/">Sphinx</a>
      1.6.3.

  </footer>

  
  </body>
</html>